Image Processing Using Opencv¶
Install Opencv¶
- Example on MacOS
$ pip3 --version
pip 24.0 from /opt/anaconda3/lib/python3.12/site-packages/pip (python 3.12)
$ python3 --version
Python 3.12.4
$ pip3 install opencv-python
Collecting opencv-python
Downloading opencv-python-4.11.0.86.tar.gz (95.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 95.2/95.2 MB 2.7 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.21.2 in /opt/anaconda3/lib/python3.12/site-packages (from opencv-python) (1.26.4)
Building wheels for collected packages: opencv-python
Building wheel for opencv-python (pyproject.toml) ... done
Created wheel for opencv-python: filename=opencv_python-4.11.0.86-cp312-cp312-macosx_10_16_x86_64.whl size=27595148 sha256=0fc6f3f12d5b5f0a5a54d870de9e866aaf59484b11dfcce77a8aece5a4f393e9
Stored in directory: /Users/bshen2/Library/Caches/pip/wheels/be/bd/d5/425eca52f204ab4b1ad7ac23c79e7a0458ee178056e4350265
Successfully built opencv-python
Installing collected packages: opencv-python
Successfully installed opencv-python-4.11.0.86
Filtering through Convolution¶
Let's use convolution for the following image processing:
- Edge Detection
- Feature Extraction
- Blurring → Scaling
Import resources and display image¶
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import cv2
import numpy as np
%matplotlib inline
# Read in the image
image = mpimg.imread('building2.jpg')
#image = mpimg.imread('test.png')
plt.imshow(image)
<matplotlib.image.AxesImage at 0x20304041c10>
Convert the image to grayscale¶
# Convert to grayscale for filtering
gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
plt.imshow(gray, cmap='gray')
<matplotlib.image.AxesImage at 0x20309fad5b0>
Convolution Through a Kernel¶
Image convolution is a mathematical operation where a small matrix (called a kernel or filter) slides over an image, performing element-wise multiplications and summing the results to produce a new pixel value. This process helps extract features such as edges, textures, and patterns by emphasizing specific spatial structures in the image. Convolution is widely used in image processing and deep learning, particularly in convolutional neural networks (CNNs) for feature detection.
Here we practice applying image convolution using traditional image processing operations.
Select a Kernel (Filter): Choose a small matrix (e.g., 3x3 or 5x5) with predefined values for a specific operation (e.g., edge detection, blurring).
Slide the Kernel Over the Image: Move the kernel across the image, covering one region at a time.
Compute the Element-Wise Product: Multiply each value in the kernel with the corresponding pixel values in the image region.
Sum the Products: Add up all the multiplied values to obtain a single new pixel value.
Store the Result: Place the computed value into the corresponding location in the output image.
Repeat for Entire Image: Continue the process by shifting the kernel until every pixel has been processed.
Here is a common 3×3 kernel for edge detection, which detects edges in a specific direction, vertical and horizontal:
Vertical Edge Detection Kernel:
$
K_x = \begin{bmatrix} -1 & 0 & 1 \\ -1 & 0 & 1 \\ -1 & 0 & 1 \end{bmatrix} $
Horizontal Edge Detection Kernel:
$
K_y = \begin{bmatrix} -1 & -1 & -1 \\ 0 & 0 & 0 \\ 1 & 1 & 1 \end{bmatrix} $
These kernels are convolved with an image to highlight vertical and horizontal edges, respectively.
Applying a filter like this to an image is a way of taking (an approximation) of the derivative of the image in the x or y direction, separately.
# Create a custom kernel
# 3x3 array for edge detection
horizon = np.array([[ -1, -1, -1],
[ 0, 0, 0],
[ 1, 1, 1]])
## TODO: Create and apply a vertical edge detection operator
vertical = np.array([[ -1, 0, 1],
[ -1, 0, 1],
[ -1, 0, 1]])
diag_45 = np.array([[ -2, -1, 0],
[ -1, 0, 1],
[ 0, 1, 2]])
diag135 = np.array([[ 0, -1, -2],
[ 1, 0, -1],
[ 2, 1, 0]])
fig = plt.figure(figsize=(24,24))
# Filter the image using filter2D, which has inputs: (grayscale image, bit-depth, kernel)
filtered_image = cv2.filter2D(gray, -1, horizon)
fig.add_subplot(2,2,1)
plt.imshow(filtered_image, cmap='gray')
plt.title('horizontal')
filtered_image2 = cv2.filter2D(gray, -1, vertical)
fig.add_subplot(2,2,2)
plt.imshow(filtered_image2, cmap='gray')
plt.title('vertical')
filtered_image3 = cv2.filter2D(gray, -1, diag_45)
fig.add_subplot(1,2,1)
plt.imshow(filtered_image3, cmap='gray')
plt.title('diag_45')
filtered_image4 = cv2.filter2D(gray, -1, diag135)
fig.add_subplot(1,2,2)
plt.imshow(filtered_image4, cmap='gray')
plt.title('diag135')
plt.show()
# Create a custom kernel for blurring
# 2x2 kernel for averaging blurring
S2x2 = np.array([[ 1, 1],
[ 1, 1]])
S3x3 = np.ones((3, 3))/9 #just based off of the 2x2 example
S5x5 = np.ones((5, 5))/25
fig = plt.figure(figsize=(48, 12))
fig.add_subplot(4,1,1)
plt.imshow(gray, cmap='gray')
plt.title('original')
# Filter the image using filter2D, which has inputs: (grayscale image, bit-depth, kernel)
blurred_image = cv2.filter2D(gray, -1, S2x2/4.0)
fig.add_subplot(4,1,2)
plt.imshow(blurred_image, cmap='gray')
plt.title('2x2')
# TODO: blur image using a 3x3 average
fig.add_subplot(4,1,3)
blurred_image_3 = cv2.filter2D(gray, -1, S3x3)
plt.imshow(blurred_image_3, cmap='gray')
plt.title('3x3')
# TODO: blur image using a 5x5 average
fig.add_subplot(4,1,4)
blurred_image_5 = cv2.filter2D(gray, -1, S5x5)
plt.imshow(blurred_image_5, cmap='gray')
plt.title('5x5')
plt.tight_layout() #so it looks nicer
plt.show()
TODO¶
Other image processing/filtering you can try:
- Other Edge Detector (e.g. Sobel Operator)
A common 3×3 kernel for edge detection is the Sobel operator, which detects edges in a specific direction. Below are the Sobel kernels for detecting vertical and horizontal edges:
Vertical Edge Detection Kernel:
$
K_x = \begin{bmatrix} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{bmatrix} $
Horizontal Edge Detection Kernel:
$
K_y = \begin{bmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ 1 & 2 & 1 \end{bmatrix} $
These kernels are convolved with an image to highlight vertical and horizontal edges, respectively.
- Corner Detection (use the kernels we discussed in slides)
- Scaling (after the blurring, can you pick one pixel out of the following?)
- 2x2
- 4x4
- Use other images of your choice
- For a challenge, see if you can put the image through a series of filters: first one that blurs the image (takes an average of pixels), and then one that detects the edges.
#sobel edge detection kernels
# used earlier edge detection to do cells and replaced kernels with sobel kernels
sobel_horizon = np.array([[ -1, -2, -1],
[ 0, 0, 0],
[ 1, 2, 1]])
sobel_vertical = np.array([[ -1, 0, 1],
[ -2, 0, 2],
[ -1, 0, 1]])
fig = plt.figure(figsize=(24,24))
filtered_image = cv2.filter2D(gray, -1, sobel_horizon)
fig.add_subplot(2,2,1)
plt.imshow(filtered_image, cmap='gray')
plt.title('sobel horizontal')
filtered_image2 = cv2.filter2D(gray, -1, sobel_vertical)
fig.add_subplot(2,2,2)
plt.imshow(filtered_image2, cmap='gray')
plt.title('sobel vertical')
plt.show()
#corner detection
kernels = [
np.array([[0, 1, 0],
[1, 1, 0],
[0, 0, 0]]),
np.array([[0, 1, 0],
[0, 1, 1],
[0, 0, 0]]),
np.array([[0, 0, 0],
[0, 1, 0],
[1, 1, 0]]),
np.array([[0, 0, 0],
[0, 1, 0],
[0, 1, 1]])
]
corner_response = np.zeros_like(gray)
for k in kernels:
response = convolve2d(gray, k, mode='same', boundary='symm')
corner_response = np.maximum(corner_response, response)
corner_response = (corner_response - corner_response.min()) / (corner_response.ptp())
threshold = 0.9
y, x = np.where(corner_response > threshold)
plt.figure(figsize=(10, 6))
plt.imshow(gray, cmap='gray')
plt.scatter(x, y, c='red', s=6)
plt.title("Corner Detection")
plt.show()
#scaling
def downscale(img, factor):
return img[::factor, ::factor]
downscaled_2 = downscale(gray, 2)
downscaled_4 = downscale(gray, 4)
plt.figure(figsize=(48, 48))
plt.subplot(4, 3, 1)
plt.imshow(gray, cmap='gray')
plt.title("Original")
plt.subplot(4, 3, 2)
plt.imshow(downscaled_2, cmap='gray')
plt.title("Downscale 2x2")
plt.subplot(4, 3, 3)
plt.imshow(downscaled_4, cmap='gray')
plt.title("Downscale 4x4")
#plt.tight_layout()
plt.show()
#series of filters
S3x3 = np.ones((3, 3))/9 #just based off of the 2x2 example
fig = plt.figure(figsize=(48, 12))
fig.add_subplot(4,1,1)
plt.imshow(gray, cmap='gray')
plt.title('original')
#blur image using a 3x3 average
fig.add_subplot(4,1,2)
blurred_image_3 = cv2.filter2D(gray, -1, S3x3)
plt.imshow(blurred_image_3, cmap='gray')
plt.title('3x3 blur')
#sobel edge detection kernels
# used earlier edge detection to do cells and replaced kernels with sobel kernels
sobel_horizon = np.array([[ -1, -2, -1],
[ 0, 0, 0],
[ 1, 2, 1]])
sobel_vertical = np.array([[ -1, 0, 1],
[ -2, 0, 2],
[ -1, 0, 1]])
filtered_image = cv2.filter2D(blurred_image_3, -1, sobel_horizon)
fig.add_subplot(4,1,3)
plt.imshow(filtered_image, cmap='gray')
plt.title('3x3 blur + sobel horizontal')
filtered_image2 = cv2.filter2D(blurred_image_3, -1, sobel_vertical)
fig.add_subplot(4,1,4)
plt.imshow(filtered_image2, cmap='gray')
plt.title('3x3 blur + sobel vertical')
plt.tight_layout() #so it looks nicer
plt.show()